FEAT Add TargetCapabilities with supports_multi_turn and adapt attacks accordingly#1433
Conversation
…rgets - Add supports_multi_turn property to PromptTarget (False) and PromptChatTarget (True) - Override to True for stateful non-chat targets (Realtime, Playwright, WebSocket) - Override to False for single-turn OpenAI targets (Image, Video) with _validate_request checks - Add _rotate_conversation_for_single_turn_target helper in MultiTurnAttackStrategy - Integrate rotation in RedTeamingAttack before sending to objective target - Adapt TAP duplicate() to skip history duplication for single-turn targets - Add ValueError guards in Crescendo, ChunkedRequest, MultiPromptSending for single-turn targets - Add unit tests for property values and attack behaviors Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…I targets OpenAIImageTarget, OpenAIVideoTarget, OpenAITTSTarget, and OpenAICompletionTarget now explicitly return False from supports_multi_turn, overriding the True inherited from PromptChatTarget via OpenAITarget. This ensures the rotation helper activates immediately, without waiting for PR 1419 to change the base class. Also fixes test assertions to match the corrected property values. Verified end-to-end: RedTeamingAttack with OpenAIImageTarget runs successfully with conversation rotation across 2 turns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rgets - Add supports_multi_turn property to PromptTarget (False) and PromptChatTarget (True) - Override to True for stateful non-chat targets (Realtime, Playwright, WebSocket) - Override to False for single-turn OpenAI targets (Image, Video) with _validate_request checks - Add _rotate_conversation_for_single_turn_target helper in MultiTurnAttackStrategy - Integrate rotation in RedTeamingAttack before sending to objective target - Adapt TAP duplicate() to skip history duplication for single-turn targets - Add ValueError guards in Crescendo, ChunkedRequest, MultiPromptSending for single-turn targets - Add unit tests for property values and attack behaviors Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…I targets OpenAIImageTarget, OpenAIVideoTarget, OpenAITTSTarget, and OpenAICompletionTarget now explicitly return False from supports_multi_turn, overriding the True inherited from PromptChatTarget via OpenAITarget. This ensures the rotation helper activates immediately, without waiting for PR 1419 to change the base class. Also fixes test assertions to match the corrected property values. Verified end-to-end: RedTeamingAttack with OpenAIImageTarget runs successfully with conversation rotation across 2 turns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR introduces a first-class capability flag (supports_multi_turn) on prompt targets so multi-turn attacks can adapt their conversation handling when interacting with fundamentally single-turn targets (e.g., image/video/TTS/completions).
Changes:
- Add
supports_multi_turnto the prompt target hierarchy (defaultFalse,Truefor chat targets, with explicit overrides for specific targets). - Adapt multi-turn attacks to rotate or avoid conversation history for single-turn targets, and add guards for attacks that require multi-turn state.
- Add unit tests covering target capability values and attack behavior/guards.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/target/test_supports_multi_turn.py | Verifies supports_multi_turn values across target classes. |
| tests/unit/executor/attack/multi_turn/test_supports_multi_turn_attacks.py | Tests conversation rotation helper and single-turn incompatibility guards. |
| pyrit/prompt_target/common/prompt_target.py | Adds supports_multi_turn default property on base target. |
| pyrit/prompt_target/common/prompt_chat_target.py | Overrides supports_multi_turn=True for chat targets. |
| pyrit/prompt_target/openai/openai_image_target.py | Marks image target single-turn; adds conversation-length safety check. |
| pyrit/prompt_target/openai/openai_video_target.py | Marks video target single-turn; adds conversation-length safety check. |
| pyrit/prompt_target/openai/openai_tts_target.py | Marks TTS target single-turn. |
| pyrit/prompt_target/openai/openai_completion_target.py | Marks completions target single-turn. |
| pyrit/prompt_target/openai/openai_realtime_target.py | Marks realtime target as multi-turn capable. |
| pyrit/prompt_target/playwright_target.py | Marks Playwright target as multi-turn capable. |
| pyrit/prompt_target/playwright_copilot_target.py | Marks Playwright Copilot target as multi-turn capable. |
| pyrit/prompt_target/websocket_copilot_target.py | Marks WebSocket Copilot target as multi-turn capable. |
| pyrit/executor/attack/multi_turn/multi_turn_attack_strategy.py | Adds _rotate_conversation_for_single_turn_target helper. |
| pyrit/executor/attack/multi_turn/red_teaming.py | Rotates conversation_id per turn for single-turn targets. |
| pyrit/executor/attack/multi_turn/tree_of_attacks.py | Avoids history duplication for single-turn targets (fresh conversation_id). |
| pyrit/executor/attack/multi_turn/multi_prompt_sending.py | Raises on single-turn targets in _setup_async. |
| pyrit/executor/attack/multi_turn/crescendo.py | Raises on single-turn targets in _setup_async. |
| pyrit/executor/attack/multi_turn/chunked_request.py | Raises on single-turn targets in _setup_async. |
9afa84a to
079751e
Compare
…otation in ChunkedRequest - Replace property overrides with _DEFAULT_SUPPORTS_MULTI_TURN class constants on all target subclasses (image, video, tts, completion, realtime, playwright, playwright_copilot, websocket_copilot) - Make supports_multi_turn settable per-instance via constructor parameter, propagated through PromptChatTarget and OpenAITarget init chains - Add supports_multi_turn to _create_identifier() params - Use self._logger instead of module logger in rotation helper - Fix video target _validate_request to use text_piece.conversation_id - ChunkedRequest: replace ValueError guard with rotation (Crucible CTF use case) - Update tests: add constructor override tests, remove ChunkedRequest ValueError test, fix PromptTarget default test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lti-turn # Conflicts: # pyrit/prompt_target/openai/openai_image_target.py # pyrit/prompt_target/openai/openai_video_target.py
Implement rlundeen2's design feedback (comment 7) to use a TargetCapabilities dataclass instead of individual class constants/properties: - Add TargetCapabilities frozen dataclass in prompt_target/common/target_capabilities.py with supports_multi_turn field (extensible for future capabilities like editable_history, json_schema_support, system_message_support, etc.) - PromptTarget: replace _DEFAULT_SUPPORTS_MULTI_TURN with _DEFAULT_CAPABILITIES, build per-instance capabilities from class defaults + constructor overrides using dataclasses.replace() - Add capabilities property for full TargetCapabilities access - Keep supports_multi_turn as convenience property delegating to capabilities - Update all subclasses to use _DEFAULT_CAPABILITIES pattern - Export TargetCapabilities from pyrit.prompt_target - Add tests for capabilities property and constructor overrides Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…structor args Replace individual supports_multi_turn kwargs on subclass constructors with a TargetCapabilities object approach: - Remove supports_multi_turn param from PromptChatTarget and OpenAITarget __init__ - PromptTarget.__init__ accepts capabilities: Optional[TargetCapabilities] for custom subclasses that call super().__init__() directly - Add capabilities property setter for per-instance overrides on any target - Update tests to use capabilities setter pattern Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…cendo error - TAP duplicate(): duplicate system messages into new conversation for single-turn targets so prepended conversation system prompts are preserved - Rotation helper: same fix - duplicate system messages when rotating conversation_id for single-turn targets instead of using bare uuid4() - Crescendo: update error message to reflect permanent incompatibility with single-turn targets (not 'does not yet support') - Update test to match new Crescendo error message Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 34 out of 38 changed files in this pull request and generated 7 comments.
Comments suppressed due to low confidence (3)
pyrit/score/float_scale/azure_content_filter_scorer.py:1
asyncio.iscoroutinefunction(api_key)won’t catch async-callable instances (objects with an async__call__). That means an async token provider may slip through and later be invoked synchronously viaTokenProviderCredential, likely returning a coroutine instead of a token string. Consider detectingiscoroutinefunction(getattr(api_key, "__call__", api_key))(or usinginspect.iscoroutinefunction) and/or validating the return type when calling the provider.
pyrit/score/float_scale/azure_content_filter_scorer.py:1- The scorer now (a) rejects async token providers and (b) automatically falls back to Entra token provider when no API key is provided. Adding tests for these behaviors would make the change safer (e.g., ValueError for async provider, fallback to token provider when
api_keyis missing/empty).
tests/unit/target/test_supports_multi_turn.py:1 - The test name implies it’s validating
PromptChatTarget, but it instantiatesMockPromptTarget, which is ambiguous from the test alone. Consider renaming the test to reflect the concrete type under test (or rename the mock to make it clear it’s a chat target), so the intent is self-evident when the test fails.
You can also share your feedback on Copilot code review. Take the survey.
- Add TestSystemPromptCarryoverOnRotation: verifies system messages are duplicated into new conversations on rotation, preserved across multiple rotations, and that user/assistant messages are not carried over - Add TestTAPNodeDuplicateSystemMessages: verifies TAP's duplicate() method copies only system messages for single-turn targets, full conversation for multi-turn targets, and always fully duplicates adversarial chat - Add ChunkedRequestAttack ValueError guard test for single-turn targets - Remove dead _rotate_conversation call in chunked_request.py (guarded by ValueError in _setup_async) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Multiple system messages (different sequences) all carried over - Empty conversation (zero messages) yields fresh conversation_id - Only-system-messages conversation preserved correctly - TAP duplicate node identity (parent_id, node_id) - TAP _conversation_context copied to duplicate - System message content preserved exactly (multiline, special chars) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Multi-piece system message (same sequence) fully duplicated - Old/original conversation untouched after rotation/duplication - Tests for both rotation helper and TAP duplicate() Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 34 out of 38 changed files in this pull request and generated 3 comments.
You can also share your feedback on Copilot code review. Take the survey.
- Test single-turn target: system prompt survives across 3 depth levels of branching (duplicate → add turn → duplicate again) - Test multi-turn target: full conversation history preserved across branching with accumulated turns - Uses real in-memory SQLite database for end-to-end verification Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The property is no longer meant to be overridden by subclasses. Multi-turn support is declared via _DEFAULT_CAPABILITIES or the capabilities constructor parameter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Test _ensure_async_token_provider function (5 tests) and OpenAITarget.__init__ API key resolution (8 tests) covering: - Explicit string key, env var fallback, param precedence - Non-Azure endpoint without key raises ValueError - Azure endpoint falls back to Entra (get_azure_openai_auth) - Sync callable wrapped in async, async callable passed through Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 35 out of 39 changed files in this pull request and generated 1 comment.
You can also share your feedback on Copilot code review. Take the survey.
…erScorer Add inspect.isawaitable check on the return value of callable api_key providers to catch cases like lambda: async_fn() that bypass asyncio.iscoroutinefunction. Closes the coroutine to prevent warnings and raises ValueError with a clear message. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 36 out of 40 changed files in this pull request and generated 10 comments.
You can also share your feedback on Copilot code review. Take the survey.
Re-run to pick up error data type fix, removing the 'Multimodal data type error is not yet supported' traceback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Video notebook requires AZURE_SPEECH_KEY and ffmpeg which are pre-existing environment dependencies unrelated to this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Clean run with no auth failures or multimodal data type errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 38 out of 40 changed files in this pull request and generated 1 comment.
You can also share your feedback on Copilot code review. Take the survey.
get_info_async() internally re-runs initialize_async() in a sandbox, which creates real Azure targets. The auth mocks must still be active when get_info_async is called, otherwise the test makes real network calls and fails without az login. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Problem
Some targets (e.g., OpenAIImageTarget, OpenAIVideoTarget) are fundamentally single-turn — they process one prompt at a time and don't use conversation history. Multi-turn attacks like RedTeamingAttack
reuse the same conversation_id across turns, which causes failures when targets validate that no prior messages exist.
There was no formal mechanism for targets to declare single vs. multi-turn support, and no way for attacks to adapt behavior accordingly.
Solution
Introduce a frozen TargetCapabilities dataclass (pyrit/prompt_target/common/target_capabilities.py) with an extensible design for future capabilities (e.g., editable_history, json_schema_support). The
first capability is supports_multi_turn: bool.
constructor parameter for per-instance overrides; capabilities are immutable after construction.
RealtimeTarget inherits from PromptChatTarget.
conversation history.
context)
_rotate_conversation_for_single_turn_target on MultiTurnAttackStrategy base class:
Testing
Entra fallback, sync/async callable handling)
Related